Transient Reward Approximation for Continuous-Time Markov Chains

A fast algorithm for the transient reward distribution in continuous-time Markov chains

This note presents a generally applicable discretization method for computing the transient distribution of the cumulative reward in a continuous-time Markov chain. A key feature of the algorithm is an error estimate for speeding up the calculations. The algorithm is easy to program and is numerically stable. c © 2000 Elsevier Science B.V. All rights reserved.

Distributions of Reward Functions on Continuous-time Markov Chains

We develop algorithms for the computation of the distribution of the total reward accrued during [0, t) in a finite continuous-parameter Markov chain. During sojourns, the reward grows linearly at a rate depending on the state visited. At transitions, there can be instantaneous rewards whose values depend on the states involved in the transition. For moderate values of t, the reward distributio...

Continuous Time Markov Chains

Discrete-time Markov chains are useful in simulation, since updating algorithms are easier to construct in discrete steps. They can also be useful as crude models of physical, biological, and social processes. However, in the physical and biological worlds time runs continuously, and so discrete-time mathematical models are not always appropriate. This is especially true in population biology –...

Continuous-time Markov Chains

Model Checking Continuous-Time Markov Chains by Transient Analysis

The verification of continuous-time Markov chains (CTMCs) against continuous stochastic logic (CSL) [3, 6], a stochastic branchingtime temporal logic, is considered. CSL facilitates among others the specification of steady-state properties and the specification of probabilistic timing properties of the form P⊲⊳p(Φ1 U I Φ2), for state formulas Φ1 and Φ2, comparison operator ⊲⊳, probability p, an...